Skip to content

Conversation

DavidLandup0
Copy link
Collaborator

@DavidLandup0 DavidLandup0 commented Jan 14, 2025

Adds DETR backbone, object detector and preprocessor.

DETR Elements

  • DETR layers
    • Positional embeddings
    • Frozen BatchNormalization
    • DETRTransformerEncoder
    • DETRTransformerDecoder
    • etc.
  • DetrBackbone (ResNet encoder)
  • Object detection preprocessor
  • DETRObjectDetector
  • Automatic conversion comes from converter API (i.e. live loading huggingface weights and porting in memory)

Preset Loader

To enable the from_preset() convention, we also convert the DETRImageConverter parameters, but it looks like the method so far has just returned None with a TODO for PaliGemma. In this PR, I've added a convert_image_converter() method to convert_detr.py and updated the general preset loader to check if a specific implementation has the method and returns the class if it does for backwards compatability.

@DavidLandup0
Copy link
Collaborator Author

Getting there :)
Lots of refactoring to do, but the core modules are there and work nicely with each other.

image

@sachinprasadhs sachinprasadhs added the WIP Pull requests which are work in progress and not ready yet for review. label Apr 11, 2025
@sachinprasadhs sachinprasadhs moved this to In Progress in KerasHub Jul 16, 2025
@sachinprasadhs
Copy link
Collaborator

@DavidLandup0 , Any update on this PR?

@DavidLandup0
Copy link
Collaborator Author

@DavidLandup0 , Any update on this PR?

Picking it up now that a couple of other things are out the way - should have a more concrete update soon :)
The architecture ~matched last time I worked on this, so not much should be left.

@DavidLandup0 DavidLandup0 marked this pull request as ready for review October 23, 2025 16:23
@sachinprasadhs sachinprasadhs self-requested a review October 23, 2025 16:24
@DavidLandup0 DavidLandup0 changed the title [Object Detection] Add DETR Model [Object Detection] Add DETR Backbone, Object Detector, Preprocessor and Weight Conversion Oct 23, 2025
@github-actions github-actions bot added the Gemma Gemma model specific issues label Oct 23, 2025
return self.converter.convert_tokenizer(cls, self.preset, **kwargs)

def load_image_converter(self, cls, **kwargs):
# Check if the converter has an image converter conversion function
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding this for backwards compatability - looks like we just didn't have a way to return the image converter so far, unless I missed it from another model.

This way, any convert_x.py that has convert_image_converter() method can return that image converter too.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Gemma Gemma model specific issues stat:awaiting response from contributor WIP Pull requests which are work in progress and not ready yet for review.

Projects

Status: In Progress

Development

Successfully merging this pull request may close these issues.

2 participants